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Criteria  for  Constructing  and  Using  an  Ada  Embedded 
System  Testbed 

AMract.  The  purpose  o(  this  report  is  to  Hst  some  of  the  criteria  used  in  five  aspects 
of  the  project:  the  hardware  configuration,  the  software  configuration,  the  real-time 
application,  the  Ada  real-time  experiments,  and  the  benchmarking  and  instnjmentation 
techniques.  Each  criterion  will  include  a  rationale.  Each  of  the  criteria  listed  in  this 
report  will  be  categorized  as  either  essential,  highly  desirable,  or  desirably 

1.  Introduction 

The  Ada  Embedded  Systems  Testbed  (AEST)  Project  was  initiated  in  October  1986  and  will  be 
continued  in  1967-88.  The  purpose  of  the  AEST  Project  is  to  investigate  some  of  the  criticai 
issues  in  using  Ada  for  real-time  embedded  applications,  particularly  the  extent  and  quality  of  the 
run-time  support  facility  provided  by  Ada  implementations.  The  Ada  run-time  is  an  exscution 
environment  that  provides  services  such  as  process  management,  storage  management,  and 
exception  handling  for  supporting  the  execution  of  Ada  programs.  These  services  were,  in  the 
past,  provided  either  by  the  application  programmer  or  by  a  smaN  real-time  executive. 

One  project  objective  is  to  ooliect,  classify,  track,  and  disseminate  information  about  the  use  of 
Ada  in  real-time  embedded  systems.  This  w«  he0  to  make  the  SEI  a  center  of  expertise  tor  such 
information.  Another  objective  is  to  create  and  expand  a  general  testbed  for  experimentation. 
The  testbed  must  acoonwnodate  different  target  processors,  different  compilers,  and  different 
toolsets.  It  must  be  designed  to  be  flexl)le,  reoonfigurable.  and  evolvable.  There  should  be  both 
harchvare  and  software  measurement  techniques  so  that  performance  data  can  be  independently 
collected  and  verified  m  a  non-intrusive  manner.  Finally  R  is  an  objective  of  the  project  to  gener¬ 
ate  new  information  about  using  Ada  in  real-time  embedded  systems.  This  information  would  be 
in  the  form  of  benchmark  test  resuRs,  higher  level  experiment  results,  and  lessons  learned  in 
designing  and  implementing  real  applications  in  Ada. 

Work  to  date  has  oonoentrated  on  identifying  the  appropriate  issues  to  investigate,  assembling 
some  of  the  necessary  harcKvare  and  software  for  the  testbed,  and  acquiring  some  of  the  existing 
benchmark  test  suRes.  A  summary  description  of  the  benchmarks  currently  available  and  an 
evaluation  of  their  appBcabIRty  to  the  AEST  Project  is  contained  in  a  report  entitled  A  Summary  of 
ffeaZ-Time  Pwtormanoa  Benchmarks  lor  the  Ada  Programming  Language  [Donohoe  87].  A  re¬ 
port  entitled  Ada  lor  Embedded  Systeme:  Issues  and  Questions  [Weidennan  87]  contains  issues 
related  to  the  use  of  Ada  language,  the  facilHies  provided  by  the  Ada  njn-time  system,  and  the 
tools  required  for  system  development. 

The  purpose  of  this  report  is  to  list  some  of  ttie  crReria  used  in  five  aspects  of  the  project;  the 
hardware  configuration,  the  software  configuration,  the  real-time  application,  the  Ada  real-time 
experiments,  and  the  benchmarking  and  instrumentation  techniques.  Each  criterion  will  include  a 
rationale.  Each  of  the  crReria  listed  in  this  report  wHI  be  categorized  as  eRher  essential,  highly 
desirable,  or  desirable.  Essential  crReria  are  those  which,  when  not  met,  wiR  severely  impact  the 
purpose  and  objectives  of  the  AEST  Project.  Highly  desirable  crReria  are  those  which,  when  not 


met,  win  limit  the  value  or  extent  of  the  results  that  are  obtained.  Desirable  criteria  are  those 
which,  when  not  met,  may  adversely  impact  the  productivity  of  the  project  personnel  or  the  cost  of 
the  project,  but  do  not  severely  impact  the  results. 

It  should  be  recognized  that  it  may  be  difficuR  or  impossbie  to  meet  many  of  the  crReria  defined  in 
this  report.  Some  of  the  crReria  may  be  conflicting  to  one  degree  or  another.  Furthermore,  all  the 
crReria  are  not  at  the  same  level  of  detail.  For  completeness,  general  as  well  as  specRic  criteria 
have  been  included  so  that  some  of  the  crReria  are  overlapping.  The  crReria  have  been  ordered 
wRhin  each  group  from  the  essential  crReria  to  the  desirable  criteria. 


2.  Criteria  for  the  Hardware  Configuration 

One  of  the  components  of  the  testbed  is  the  hardware.  It  includes  the  target  computer  as  well  as 
the  ancillary  processors,  connectors,  I/O  equipment,  and  test  equipment  necessary  to  carry  out 
experimentation.  The  purpose  of  this  sect'nn  is  to  list  the  criteria  tor  the  suRe  of  hardware  needed 
for  the  AEST  Project. 

Crfterlon  HW1  :  The  generic  testbed  hardware  must  support  compilation  of  Ada  programs, 
downloading  of  Ada  programs  to  a  targM  system,  simulation  of  the  environ¬ 
ment  for  a  target  system,  and  monRoring  of  Ada  programs  running  in  the 
target  environment. 

Rationale:  These  are  basic  activRies  of  software  development  for  mission-crRical  com¬ 

puter  resource  (MCCR)  systems. 

Criticality:  Esserrtial 


Criterion  HW2 : 

Rationale: 

Criticality: 


The  target  systems  should  be  representative  of  those  which  are  presently 
used  in  MCCR  applications  or  for  which  future  use  is  anticipated  or  desirable. 

We  must  address  the  needs  of  the  DoO. 

Essential 


Crfterlon  HW3 : 


Rationale: 

Crftlcallty: 


There  should  be  four  interactbg  computing  systems  —  a  host  development 
system,  a  target  system,  an  environment  sirrxjlator,  and  a  monRor  system 
(see  Figure  1). 

This  configuration  provides  all  the  necessary  functionalRy  and  allows  the 
flexibHRy  to  change  the  target  wRhout  changing  any  of  the  other  components. 

Highly  desirable 


Critarlon  HW4 : 

Rationale: 

Cmicallty: 


The  host  system  should  be  one  for  which  there  are  a  variety  of  cross  conpii- 
ers. 

TNs  is  required  to  leverage  the  expertise  of  the  project  group  and  to  get 
maximum  information  and  benefR  from  the  available  hardware. 

Highly  desirable 


2 


CMU/SEM7-TR-30 


Flgural:  Generic  Embedded  System  Testbed 


CrttMlon  HW5 : 
Rational*: 


Criticality: 


The  target  systems  should  be  orv  s  for  which  there  are  a  variety  of  cross 
oorrpilers. 

This  criterion  allows  the  comparison  of  different  compilers  on  identical  ar¬ 
chitectures.  The  targets  for  which  there  are  multiple  cross  compilers  also 
represent  those  targets  that  are  technologically  advanced  or  DoD  standards. 

Highly  desirable 


Criterion  HW6 : 

The  target  systems  should  represent  modem  architectures  which  are  capable 
of  supporting  the  demands  of  higher  level  languages  such  as  Ada. 

Rationale: 

We  must  be  aware  of  the  current  state-of-the-art  with  regard  to  target  proces¬ 
sors  and  their  ability  to  efficiently  handle  Ada. 

Cntlcallty: 

Highly  desirable 

Criterion  HW7 : 

The  test  equipment  should  include  a  logic  analyzer. 

Rationale: 

The  project  needs  non-intrusive  testing  capabilities  due  to  the  sensitivity  of 
the  timing  constraints.  A  logic  analyzer  provides  the  capability  to  check  the 
periodicity  of  activities,  uncover  bottlenecks,  and  check  software  timings  inde¬ 
pendent  of  the  software. 

Criticality: 

Highly  desirable 

Criterton  HW8 : 

The  hardware  testbed  should  be  flexible,  reconfigurable,  and  evolvable. 

Rationale: 

This  is  a  requirement  for  any  hardware  system  on  which  experimentation  will 
occur.  The  IEEE  Computer  special  issue  on  distributed  system  testbeds 
(Berg  82]  contains  some  general  guidelines  for  building  testbeds.  The  alter¬ 
native  of  building  separate  hardware  for  each  experiment  is  impractical  and 
expensive. 

Criticality: 

Desirable 

Criterton  HW9 : 

There  should  be  the  ability  to  replace  one  target  system  with  another  without 
changing  or  replacing  the  entire  testbed.  This  requires  that  the  interfaces 
between  the  various  systems  and  between  systems  and  10  devices  be 
reasoriably  standard  so  that  costly  devices  can  be  shared. 

Rationale: 

While  the  mechanism  for  downloading  or  interfacing  with  other  computers 
can  be  expected  to  change,  it  is  desirable  to  reuse  the  other  components  of 
the  testbed  for  multiple  targets. 

Criticality: 

Desirable 

Criticality: 


3.  Critttria  for  the  Software  Configuration 

The  software  crtteria  for  the  testbed  can  be  subdivided  into  two  categories.  The  first  is  the  Ada 
cross  cotnpiler  crtteria,  and  the  second  is  the  cross  development  environment  criteria.  The  cross 
compiler  must  satisfy  certain  minimum  criteria  to  be  usabie  in  an  embedded  system  application. 
Other  criteria  are  desirable,  but  not  essential.  The  cross  development  environment  contains  a  set 
of  tools  that  make  the  job  of  development  of  MCCR  software  easier. 

3.1.  Cross  Compiler  Criteria 

Criterton  SW1  :  The  compiler  should  be  targeted  to  MicroVAX-ll  or  MC680x0  microproces¬ 
sors. 

Ratlonaia:  These  are  the  first  two  targets  to  be  explored  by  the  project.  In  the  second 

year  this  constraint  will  be  relaxed. 

Criticality:  Highly  desirable 

Criterion  SW2  :  The  following  options  related  to  pragmas  should  be  supported: 

•  Warning  messages  for  all  unrecognized  pragmas  Referenc* 
Manual  tor  toa  Ada  Programming  Language'  [2.8(1 1 )]  [ANSI  83]. 

•  Pragma  INLINE  should  be  supported  (the  compiler  should  detect 
and  flag  any  situations  where  the  pragma  cannot  be  executed, 
for  example,  a  recursive  subprogram)  [LRM  6.3.2(4)] 

•  Pragma  INTERFACE  ^tould  be  supported  for  the  assembly  lan¬ 
guage  of  the  target  machine  and  for  other  languages  for  which 
appropriate  llxary  software  exists  [LRM  Appendix  B] 

•  Pragmas  SUPPRESS.  ELABORATE.  LIST,  and  PAGE  should 
be  supported  [LRM  Appendix  B] 

Rationale:  The  compiler  should  let  the  programmer  know  what  actions  are  being  taken. 

The  pragmas  INLINE,  SUPPRESS,  and  ELABORATE  are  necessary  for  in¬ 
vestigating  performance  issues.  Pragma  INTERFACE  is  necessary  for  op¬ 
timization  and  reusability.  LIST  and  PAGE  are  useful  for  suppressing  and 
organizing  source  code  listings. 

Criticality:  Highly  desirable 


Critarton  SW3 :  The  compiler  should  provide  SHORTJNTEGER,  LONGJNTEGER, 
SHORT_FLOAT.  and  LONG_FLOAT  types  [LRM  3.5.7(7,8)].  The  compiler 
should  provide  unsigned  data  types  UNSIGNED_LONGWORD, 
UNSIGNED.WORD,  and  UNSIGNED_BYTE. 

Rationale:  It  can  be  expected  that  most  real-time  applications  will  use  a  variety  of  short 

and  long  data  types. 

Criticality:  Highly  desirable 


'  Rtfmnag  Manual  for  Sw  Ada  Programming  Languaga  wfll  b*  ratamd  to  as  lh«  LRM  throughout  the  mt  of  this  roport. 


Crttwlon  SWA  :  The  minimum  range  on  predefined  type  PRIORITY  should  be  0  to  1 5. 

Rationale:  Multiple  levels  of  priority  are  necessary  to  support  different  scheduling 

models. 

Criticality:  Highly  desirable 


Criterion  SW5  :  The  following  features  (discussed  in  Chapter  13  of  the  LRM)  should  be  sup¬ 
ported; 

•  Represerltation  clauses  [LRM  13.1] 

•  Enumeration  representation  clauses  [LRM  13.3] 

•  Record  represerriation  clauses  [LRM  13.4] 

•  Address  clauses  (interrupts)  [LRM  13.5] 

•  Change  of  representation  [LRM  13.6] 

•  Interface  to  other  languages  (assentiler,  HOL)  [LRM  1 3.9] 

•  Unchecked  type  conversions  [LRM  13.10.1] 

•  Length  clauses  [LRM  13.2] 

•  Unchecked  storage  deallocation  [LRM  13.10.1] 

Rationale:  These  are  (precisely  the  features  that  were  inserted  into  the  ianguage  to  sup¬ 

port  embedded  computer  system  (ECS)  applications. 

Criticality:  Highly  desirable 


Criterion  SW6  :  The  compiler  should  have  low  level  I/O  packages  to  support  real-time  device 
drivers.  [LRM  14.6] 

Rationale:  This  is  a  requirement  of  ECS  applications. 

Criticality:  Highly  desirable 


Criterion  SW7  :  The  compiler  should  have  the  following  real-time  characteristics: 

•  DURATION'SMALL  less  than  100  microseconds 

•  SYSTEM  .TICK  less  than  1  millisecond 

•  Tasks  executing  a  delay  should  be  rescheduled  within 
SYSTEM.TICK  of  the  expiration  of  the  delay 

•  User  selectable  scheduling  algorithm  (preferably  by  pragma) 

Rationale:  These  are  necessitated  by  the  severe  timing  constraints  of  real-time  em¬ 

bedded  systems.  In  many  applications  there  are  timing  requirements  in  the  i 
millisecor^  range.  The  scheduling  algorithm  must  be  known  by,  if  not  con¬ 
trolled  by,  the  application  programmer. 

Highly  desirable 


Criticality: 


Crft«rlon  SW8  :  Successive  choices  of  nevv  compilers  tor  test  and  experimeniation  should  be 
based  on: 

•  the  need  to  support  a  newr  target  computer 

•  variety 

•  the  inclusion  of  one  or  hwo  compilers  likely  to  be  industry  stan¬ 
dards 

Rationale:  First  priority  is  to  support  selected  target  conputers.  Experimentation  with  a 

variety  of  compilers  will  make  the  results  more  credible  and  less  subject  to 
the  idiosyncrasies  of  a  particular  compiler.  Including  one  or  two  ‘standards* 
will  provide  stable  reference  points  for  comparison. 

Criticality:  Highly  desirable 


Crtterton  SW9 :  Tasks  dependent  on  library  packages  are  required  to  terminate.  [LRM 
9.4(13)1 

Rationale:  This  ensures  that  all  deperpent  tasks  terminate  before  the  main  program. 

Cntieallty:  Desirable 


Crtterlon  SW1 0  :  The  compiler  must  be  hosted  on  a  Micro Vax-II  running  either  the  VMS  or 
ULTRix  operating  system. 

Rationale:  This  is  a  practical  constraint  dictated  by  the  hardware  available  at  the  SEI  as 

well  as  the  experience  of  the  software  engineers  assigned  to  the  project. 
This  constraint  may  be  relaxed  as  time  passes. 

Criticality:  Desirable 


Crtterlon  SW1 1  :  The  compiler  should  be  able  to  generate  code  for  the  host  machine  and  the 
target  machine. 

Rationale:  This  permits  some  of  the  initial  unit  testing  to  be  done  on  the  host  machine 

and  frees  the  more  restricted  target  machine  for  integration  testing  and  oper¬ 
ational  testing. 

Crfticailty:  Desirable 


Crtterlon  SW1 2 :  The  compilation  speed  should  be  500  lines  per  minute  or  greater  on  a 
MicroVAX-ll.  The  disk  space  required  should  be  less  than  50  megabytes. 
The  main  memory  required  should  be  less  than  8  megabytes.  The  virtual 
memory  required  should  be  less  than  40  megabytes.  The  oin-time  resources 
should  be  less  than  20  kilobytes. 

Rationale:  These  numbers  are  known  to  be  achievable.  Numbers  not  achieving  these 

standards  are  indications  of  poor  engineering. 

Crtticaitty:  Desirable 
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CrltMlonSW13 :  Storage  should  be  reclaimed  when  an  object  becomes  inaccessible  [LRM 
4.8(7)].  Furthermore,  the  actual  time  of  reclamation  should  be  under  pro¬ 
grammer  control. 

Rationale:  Without  storage  reclamation  (garbage  collection),  there  will  be  a  tendency  to 

mn  out  of  dynamic  memory  space.  Programmer  control  is  necessary  so  that 
critical  timing  deadlines  are  not  missed. 

Criticality:  Desirable 


Criterion  SW1 4 : 

Rationale: 

Criticality: 

The  only  restrictions  on  the  main  program  (which  appear  in  Appendix  F  of  the 
LRM)  should  be  that  the  main  pr^ram  is  a  subprogram  without  parameters. 
In  case  of  a  function,  the  result  type  should  be  a  discrete  type  [LRM  10.1(8)]. 

In  an  embedded  computer  system  there  should  be  no  operating  system,  and 
the  mechanism  for  invoking  a  main  program  should  be  as  simple  as  posstole. 

Desirable 

Criterion  SW15 : 

The  performance  of  the  Ada  run-time  system  should  satisfy  ttte  folowing  re¬ 
quirements: 

•  Interrupt  latency  less  than  lOO  mfcroseconds 

•  Overhead  for  simple  task  switch  less  than  200  microeeoonds 

•  Overhead  for  simple  task  rendezvous  less  than  200 
microseconds 

Rationale: 

Fast  context  switches  are  required  by  tfie  severe  timing  oonstraires  of  real¬ 
time  systems.  Failure  to  meet  these  performance  standards  wil  encourage 
workarounds  representing  poor  software  engineering  practcM 

Criticality: 

Desirable 

Criterion  SW1 6 : 

The  compilation  system  should  have  a  sophiaticaiad  library  management 
system.  Among  the  functions  to  be  supported  are  creation  and  deletion  oi 
p^ram  libraries,  sharing  of  program  Riraries,  interrogation  of  program 
libraries  (list  unit  name  and  type,  list  unit  dependencies,  determine  complete¬ 
ness,  determine  recompilation  order),  and  mariipulation  of  program  libraries 
(removing  a  compilation  unit,  clearing  the  entire  library). 

Rationale: 

These  are  basic  functions  of  a  library  management  system  of  an  Ada  compi¬ 
lation  system. 

Criticality: 

Desirable 

Ci1tarionSW17 : 

The  compilation  system  should  support  compilation  management  activities 
including  batch  and  interactive  completion  modes,  listing  of  compilation  units 
obsolesced  by  compiling  another  unit,  and  automatic  recompilation  of  units 
obsolesced  by  oorrpiiing  another  unit. 

Rationale: 

These  functions  make  the  software  engineer  much  more  productive  in  devel¬ 
oping  an  application. 

Criticality: 

Desirable 
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CrHarlon  8W18 :  Th«  oofnpilation  system  should  have  comprehensive  documentation.  This 
should  Include  programming  restricttons  and  known  bugs. 

Rationale:  The  system  ntay  be  dlfflcuR  to  use  without  good  documentation. 

CiflleaRly:  Desirable 


Ciltarion  SW19  :  The  oompMatton  system  should  have  intormative  diagnostic  (error)  messages. 

Rationale:  This  function  makes  the  software  engineer  more  productive  in  deveioping  an 

appNcatbn. 

CrltlcaRty:  Oesirabie 


Ciltailon  SW20  :  The  compilation  system  should  dearly  document  all  implementation  depend¬ 
ent  charaderistics  [LRM  Appendix  F]  induing; 

•  The  form,  alowed  piaces,  and  effed  of  every  impiementation- 
dependent  pragma  (LRM  2.8(8)] 

•  The  status  of  each  language  defined  (standard)  pragma:  sup¬ 
ported  or  unsupported  by  the  implemenlation 

•  The  name  and  type  of  every  implementation-dependent  altrtiute 
(LRM  4.1 .4(4)] 

•  The  spedfication  of  the  package  SYSTEM  (LRM  13.7(1)] 

•  A  list  of  aR  restrldions  on  representation  clauses  (LRM  13.1(10)] 

•  The  conventions  used  for  any  implemertation-generated  names 
denoting  impiementatiorvdependent  components  (LRM  13.4] 

•  The  interpretation  of  expressions  that  appear  in  address  clauses, 
indudbig  those  for  interropts  (LRM  13.5(3)] 

•  Any  restrtetione  on  unchecked  conversions  (LRM  13.10.2] 

•  Any  implementalion-dependent  charaderistics  of  the  input- 
output  packages  (LRM  14.1(1),  14.1(11),  14.2.1(13),  14.4(1)] 

Rationale:  These  are  spedalzed  charaderistics  of  the  implementation  and  need  to  be 

part  of  a  comprehensive  documentation  package- 

CrfUcaMy:  Desirable 


3.2.  Cross  Dsvsiopmsnt  Environmsnt  Critsrls 

Crflarlon  8W21  :  The  cross  development  environment  must  incorporate  the  following  toois 
(see  (WekJennan  87]  for  more  detailed  descriptions): 

•  Source  code  cross  referencer 

•  Source  code  Hster  (optionai  assembly  code  interspersed  in  Ada 
listing) 

•  Ada  linker 


•  Target  load  module  (system)  txjilder 

•  Symbolic,  source  level  debugger  (rerrwte  debugging  capability) 

•  Load  module  downloader/receiver 

Rationale:  These  functions  are  all  critical  to  the  efficient  and  well-stmctured  develop¬ 

ment  of  real-time  embedded  systems. 

Crttlcellty:  Essential 


Crtterlon  SW22  :  The  cross  development  environment  should  incorporate  the  following  tools 
(see  [Weiderman  87]  for  more  detailed  descriptions); 

•  Pretty  printer 

•  Lartguage  sensitive  editor 

•  Static  profiler 

•  Frequency  analyzer 

•  Cross  assembler 

•  Target  simulator 

•  Test  manager 

•  Configuration  manager 

•  Module  manager 

Rationale:  These  functions  are  all  desirable  for  the  efficient  and  well-structured  devel¬ 

opment  of  real-time  embedded  systems. 

Criticality:  Desirable 


Crtterlon  SW23  :  There  must  be  comprehensive  documentation  and  support  for  all  the  cross 
development  environment  tools. 

Rationale:  The  usability  of  the  tools  is  directly  proportional  to  the  quality  of  their  docu¬ 

mentation  and  support. 

Criticality:  Desirable 


4.  Criteria  for  the  Real-Time  Application 

The  purpose  of  writing  a  complete  application  for  the  testbed  is  to  provide  a  vehicle  to  test  the 
Ada  language,  the  rurv-time  system,  and  the  target  processor  at  the  coarsest  level  of  granularity. 
The  application  will  provide  a  proof  of  concept  that  Ada  can  be  used  for  the  design  and  imple¬ 
mentation  of  time-critical  MCCR  applications.  It  will  also  be  a  generator  of  additional  issues, 
provide  a  context  for  the  usability  of  the  information  gained  though  experimentation,  and  provide 
a  software  engineering  exercise  for  real-time  programming  in  Ada. 
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Cfitwion  API  :  This  is  typical  ECS  functionaUty  that  needs  to  be  implenfwnied  in  Ada.  In 
partfcular,  it  should  have  strict  timing  demands,  multipie  concurrent  activities, 
low-level  I/O,  error  handling,  interrupts,  and  periodic  activities. 

Rationale:  This  is  precisely  the  environment  tor  which  Ada  was  designed.  It  is  this 

environment  tor  which  Ada's  utilNy  needs  to  be  demonstrated. 

Criticality:  Essential 


Cntarlon  AP2 :  The  application  should  be  large  enough  to  stress  the  system  and  small 
enough  to  be  feastoie  with  respect  to  the  equipment  and  personnel  resources 
available. 

Rationale:  All  the  Ada  features  and  capabilities  tor  real-time  embedded  systems  should 

be  used  in  combination  in  order  to  provide  a  tme  test  for  Ada.  The  imple¬ 
mentation  tirrw  should  not  exceed  approximately  six  months  (2-4  people)  so 
that  results  will  be  timely. 

Criticality:  Essential 


cntarlon  AP3  :  The  application  should  be  an  existing  ECS  application  or  should  be  derived 
from  an  existing  ECS  appHcation.  If  a  subset  of  an  application  is  used,  it 
should  be  a  subset  that  does  not  sinpilfy  the  functionaHty  implemented. 

Rationale:  The  application  will  be  credtole  to  the  MCCR  community  orriy  If  it  can  be 

related  to  a  "rear  application. 

Criticality:  Highly  desirable 

cntarlon  AP4  :  The  application  should  be  easily  ported  from  one  target  processor  to  another. 

This  requires  that  there  be  a  minimum  of  specialized  sensor  and  actuator 
hardware  requiring  individual  toteifaces. 

Ratlonala:  The  usefulness  of  the  application  is  enhanced  if  it  can  be  reused  across 

different  target  architectures. 

Criticality:  Highly  desirable 

cntarlon  APS  :  Thare  should  be  a  working  version  of  the  application  available. 

Ratlonala:  Tha  kinctionaiity  and  performance  of  the  working  application  (in  another 

higher  level  language  or  assembly  language)  can  be  compared  with  the  /kfa 
version  tor  lUnctionaiily  and  performance. 

CnncaNty:  Highly  desirable 


cntarlon  APS :  The  system  should  be  visually  oriented.  That  is,  there  should  be  a  graphical 
user  interface  or  devices  that  can  be  easily  controlled. 

Ratlonala:  This  criterion  makes  demonstrations  more  interesting  and  would  add  to  the 

software  showcase  of  the  SEI. 

Highly  desirable 


CnttoaHty: 


Criterion  APT : 

Rationale: 

Criticality: 

The  schedule  for  the  development  of  the  application  must  include  time  for; 

•  recording  design  decisions 

•  analyzing  alternative  designs 

•  recording  problems  encountered  in  applying  Ada  to  the  appli¬ 
cation 

•  recording  lessons  learned 

The  usefulness  of  the  application  work  will  be  enhanced  if  it  is  more  than  a 
bare  ’existence  proof.’ 

Highly  desirable 

Criterion  APS : 

The  application  should  require  a  minimum  of  domain-specific  knowledge. 

Rationale: 

The  majority  of  the  personnel  assigned  to  design  and  implemertt  the  appli¬ 
cation  will  be  software  engineers.  Knowledge  of  physics  or  Kalman  filters,  for 
example,  will  not  enhance  the  information  derived  from  the  application  with 
regard  to  the  use  of  Ada  in  embedded  systems. 

Criticality: 

Desirable 

Criterion  APS : 

There  should  be  an  industrial  or  government  sponsor  that  is  willing  and  able 
to  provide  domain-specific  knowledge  about  the  application.  Furthermore, 
there  should  be  easy  access  to  this  knowledge,  preferably  in  the  form  of  an 
affiliate  working  at  the  SEI. 

Rationale: 

To  make  the  application  relevant  and  credtole  requires  that  someone  knowl¬ 
edgeable  in  the  domain  area  work  closely  with  the  project  personnel.  This 
person  would  help  to  define  the  requirements  and  help  with  the  design. 

Criticality: 

Desirable 

CrlMrion  AP10  :  There  should  be  high  quality  documentation  available  for  the  system  being 
implemented.  The  documentation  should  include  a  system  overview,  system 
requirements  specification,  software  requirements  specification,  high  level 
design,  detailed  design,  code,  and  test  suites. 

Rationale:  The  project  is  not  particularly  concerned  with  the  early  portion  of  the  lifecycle. 

While  the  design  will  be  influenced  by  Ada,  the  proj^  should  invest  as  little 
resource  as  possible  in  defining  the  problem  and  specifying  requirements. 

Desirable 


Criticality: 


5.  Criteria  for  Ada  Reai-Time  Experiments 

The  purpose  of  the  Ada  real-time  experiments  is  to  design  and  develop  a  set  of  experiments  that 
assess  the  feastoiNty  of  implementing  essential  embedded  system  functionality  in  Ada.  The  num¬ 
ber  of  experiments  designed  should  be  reasonable  (with  respect  to  resources)  and  should  pro¬ 
vide  maximum  coverage  of  the  issues  and  questions  defined  in  [Weiderman  87]. 

Criterion  EX1  :  The  experiments  must  provide  direct  support  of  the  application  being  devel¬ 
oped  by  the  AEST  Project  as  well  as  indirect  support  to  other  similar  MCCFI 
applications. 

Rationale:  The  AEST  application  is  being  designed  to  address  many  of  the  same  prob¬ 

lems  that  exist  in  the  broader  MCCR  community.  Support  of  the  AEST  appli¬ 
cation  will  guarantee  support  of  the  mission  of  the  AEST  Project. 

Cmicallty:  Essential 


Criterion  EX2 : 

The  experiments  should  be  defined  broadly  to  examine  a  number  of  related 
issues  rather  than  a  single  issue.  They  should  be  defined  so  as  to  address 
existing  issues  as  weH  as  raise  new  issues  related  to  the  use  of  Ada  in  real¬ 
time  embedded  systems. 

Rationale: 

This  will  allow  greater  productivity  in  addressing  the  issues.  Narrowly 
focused  experiments  are  the  purpose  of  the  benchmarks  and  instmmentation 
task. 

Criticality: 

Highly  desirable 

Criterion  EX3 : 

Each  group  of  experiments  should  be  designed  with  a  clear  purpose,  descrip¬ 
tion,  approach,  set  of  measurements  to  be  perfonned,  and  results  to  be 
achieved.  All  experiments  should  be  carefully  documented. 

Rationale: 

TNs  criteria  represents  sound  experimental  methodology  and  should  be  fol¬ 
lowed  to  facltate  transition  of  useful  results. 

Criticality: 

Highly  desirable 

Criterion  EX4  :  The  artalysis  criteria  must  include  functionaiity;  i.e.,  can  a  certain  function  be 
implemented  in  a  straightforward  and  efficient  way  using  Ada?  Alternatively, 
are  there  solutions  (e.g.,  assembly  language)  that  are  straightforward  and 
efficient? 

Rationale:  Not  all  functionality  must  be  implemented  in  Ada.  That  which  is  straightfor¬ 

ward  and  efficient  in  Ada  should  be  written  in  Ada.  Assembler  should  be 
used  only  when  absolutely  necessary. 

Highly  desirable 


Criticality: 


Crfttrlon  EX5  :  Th«  analysis  criteria  should  include  objective  performance  measures.  In  par¬ 
ticular,  they  should  include  execution  speed  on  the  target  processor,  system 
load  module  size,  Ada  run-time  size,  and  source  code  size. 

Rationale:  These  are  imporiant  usability  measures  for  Ada  code  running  in  MCCR  sys¬ 

tems. 

Critleallty:  Highly  desirable 

Crltarlon  EX6  :  The  analysis  criteria  should  include  subjective  measures  of  the  source  code 
including  its  complexity,  maintainability,  readability,  and  portability. 

Rationale:  The  software  engineering  benefits  of  using  Ada  in  MCCR  systems  must  be 

evaluated  because  they  can,  in  the  long  term,  overshadow  some  of  the  pos¬ 
sible  near-term  deficiencies  of  Ada  implementations.  While  these  measures 
must  be  evaluated  subjectively  in  e)^riments  of  this  scope,  they  are  impor¬ 
tant  nevertheless.  Some  limited  objective  testing  of  portability  can  be  done 
with  the  various  targets  of  the  testbed. 

CrHIcallty:  Highly  desirable 


Crltarlon  EX7  ;  The  foUowirtg  experiment  areas  should  be  given  priority  as  areas  which  sup¬ 
port  project  application  development  of  the  AEST  Ptoje^: 

•  representation  clauses  —  bit  manipulations,  converting  machine 
representations  of  values  into  other  forms  for  exterrtal  communi¬ 
cation 

•  tasking  with  pdorities 

•  periodic  scheduling  ^  tasks  scheduled  on  basis  of  time  intervals 
(s.g.,  every  5  milliseconds) 

•  interrupt  handling  —  timer  interrupts  for  scheduling  purposes 

•  use  of  math  itorary  —  trigonometric  functions,  matrix  manipu¬ 
lation 

•  buffering  mechanisrrrs  —  shared  data  storage  used  for  intertask 
oomrrajnicalion 

•  data  trarrsfer  mechanisms  between  machines  —  data  transfer 
over  parallel  communication  channels  including  actual  data. 
prcMocd  information,  and  handling  of  time-out  conditions 

Railonalo:  These  are  appiicatiort-oriented  functions  that  can  be  expected  to  be  important 

for  the  AEST  Project. 

Criticality:  Highly  desirable 

Criterion  EX8  :  The  experiments  should  be  feasible  with  respect  to  the  equipment  and  per¬ 
sonnel  resources  available. 

Rationale:  Approximately  two  to  three  people  will  be  assigned  to  this  task.  The  breadth 

of  scope  of  the  experimentation  should  take  these  constraints,  as  well  as  the 
hardware  constraints  into  account. 

CrttlcalHy:  Desirable 
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CrIlMlon  EX9 


Ratlonato: 


Cmieallly: 


TTw  folkxwino  expeflrrMnt  areas  should  be  given  priority  as  areas  which  are 
retevara  to  application  developers  in  general; 

•  low  level  I/O  —  interrupt  hartdiing,  interfacing  to  devices 

•  concurrent  control  —  multitasking  capabilities,  scheduling  strat¬ 
egy  employed 

•  application  schedulers  —  cyclic,  event/data  driven,  periodic 

•  time  measurement  —  interrupt  latency,  context  switch  time,  pre¬ 
cision  and  overhead  of  delay 

•  time  managemera  —  package  calendar,  clock  resolution, 
SYSTEM.TICK,  OURATION’SMALL 

•  internal  represeraation  —  abaity  to  access  and  manipulate  bits 

•  error  handling 

•  pragmas  —  which  are  supported,  implementatton  semantics 

•  memory  management  —  static  araj  dynamic  allocation,  garbage 
collection 

•  numerical  computation 

These  experiment  areas  are  importara  to  application  developers  but  of  lower 
prioray  to  the  project  appkcatton.  They  should  be  harajled  as  time  and 
resources  perma. 

Desirable 


6.  Criteria  for  Benchmarks  and  Instrumentation  Techniques 

The  benchmark  tests  and  instrumentation  tectviiques  address  the  finest  level  of  granularity  for 
testing  Ada  for  real-tims  entMdded  system  appkcations.  TNs  area  concentrates  on  the  language 
features  rather  than  the  functtonaaty  required  by  the  application  programmer.  The  information 
generated  by  this  actMty  is  crucial  to  the  application  development  and  the  Ada  real-time  experi- 
meraation.  This  acUvity  must  provide  a  firm  practical  and  theoretical  foundation  on  which  other 
activities  can  draw. 


Criterion  BM1 


CnttealHy; 


Critenon  BM2 


Rationale: 


Whenever  possible,  high  quality  benchmark  test  suites  should  be  imported, 
rather  than  designed  and  implemented  at  the  SEI. 

Much  good  work  has  already  been  done.  There  is  little  point  in  reinventing 
test  technology.  Unfortunately,  there  is  also  much  poor  work  that  has  been 
done.  Protect  personnel  must  take  great  care  to  discriminate  between  the 
good  work  and  the  bad  work. 

Essential 


The  collection  and  reporting  of  test  suite  data  should  have  strong  practical 
and  theoretical  under^mngs. 

Tests  cannot  be  designed,  implemented,  mn,  or  interpreted  haphazardly. 
There  are  many  pitfalls  in  doing  benchmark  testing.  It  must  be  determined 
that  you  are  measuring  what  you  think  you  are.  (A  variety  of  purely  software. 


K? 


hardware-assisted,  and  purely  hardware  techniques  should  be  investigated 
and  compared  against  each  other.)  The  configurations  (hardware  and 
software)  must  be  tightly  controlled  and  recorded. 

Criticality:  Essential 

Criterion  BM3  :  The  time  and  space  implications  of  the  various  Ada  features  should  be  ex¬ 
amined. 

Rationale:  Ttiese  are  the  two  most  important  criteria  for  MCCR  application  builders. 

Crttleallty:  Essential 

Criterion  BM4  :  The  benchmark  tests  should  concentrate  on  individual  language  features 
rather  than  the  interactions  between  language  features. 

Rationale:  Other  activities  within  the  project  will  be  addressing  feature  interactions. 

Criticality:  Highly  desirable 

Criterion  BMS  :  There  should  be  hardware  verification  of  software  timing  results. 

Rationale:  It  has  been  our  experience  that  software  timings  are  very  elusive  because  of 

software  tirrers,  daemons,  and  strange  implementations  by  compilers.  It  has 
not  been  unusual  to  get  negative  values  when  the  efficiency  of  a  feature  is 
determined  by  taking  the  time  difference  between  a  control  program  with  a 
null  loop  and  an  experiment  program  with  a  loop  containing  the  feature. 
Hardware  verification  of  timin^^  through  the  use  of  a  logic  analyzer  is  one 
method  to  increase  the  credtoility  of  timing  results. 

Crttleallty:  Highly  desirable 

Crttarlon  BM6  :  The  test  suite  should  include  composite  as  well  as  individual  benchmarks. 

Rationale:  Composite  benchmarks  give  an  overall  figure  of  merit  based  on  a  group  of 

features.  Examples  include  the  Whetstone  and  Dhrystone  benchmark  pro¬ 
grams.  Individual  benchmarks  attempt  to  isolate  individual  features  for  test¬ 
ing.  Both  these  techniques  provide  useful  results. 

Crttleallty:  Highly  desirable 

Crttarlon  BM7  :  The  benchmark  test  suites  should  include  measurements  of  the  following  Ada 
features  (see  (Donohoe  87]  for  nx>re  details): 

•  Subprogram  calls 

•  Interrupt  laterrcy 

•  Context  switching  and  synchronization  (rendezvous) 

•  Dynamic  storage  allocation 

•  Exception  hartdling 

•  Explicit  type  conversions  (involving  representation  specs) 

•  Task  elaboration,  activation,  and  termination 


•  CLCX^K  function  overhead 

•  TIME  and  DURATION  evaluations 


Rationale: 

Criticality: 

Criterion  BM8  : 
Rationale: 

Criticality: 

Criterion  BM9  : 

Rationale: 

Criticality: 


These  are  important  Ada  features  for  real-time  embedded  system  program¬ 
ming. 

Highly  desirable 

-  ! 

Gaps  in  the  existing  benchmark  test  suites,  particularly  in  the  area  of  interrupt 
handling,  interrupt  latency,  and  context  switching,  need  to  be  filled. 

Most  of  the  existing  tests  have  been  run  only  on  host  systems  where  it  is 
difficult  to  generate  interrupts.  Thus,  there  are  few  benchmark  tests  that  deal 
with  this  important  area. 

Highly  desirable 


Testing  should  include  results  of  compiler  efficiency,  but  with  a  lower  priority. 
Run-time  efficiency  is  much  more  important  than  compile  time  efficiency. 
Desirable 
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